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Non-linear picture processing 



The present invention relates to picture signal processing and, in particular, but 
not exclusively, to adaptive improvement of picture quality. 

5 Although high quality display devices are currently available for displaying, 

for example, video and computer-generated images, limitations nevertheless still exist 
particularly with digital displays such as LCD display panels and monitors. It is therefore 
recognized that there is further room for improvement of image quality and it is appreciated 
that image quality could be noticeably improved by achieving increases in color saturation. 

10 However, display devices commonly exhibit a maximum acceptable saturation value and it 
has been found that, while some areas of a displayed image, with particular colors, might 
well be at, or near, their maximum saturation value, other areas with different colors might 
still be somewhat below such a maximum value. 

Yet, furthermore, and particularly with current digital LCD monitors, 

15 relatively few picture adjustments are available to the user. Most frequently, picture contrast 
and brightness can be adjusted, while saturation, hue and color temperature are less 
frequently available. In such digital displays, each pixel is driven by a triplet of values 
wherein each one represents the intensity of one of the three colors red, green and blue. 
However, due to the finite number of bits serving to represent each color value, the possible 

20 driving values are likewise limited. For example, in a 24-bit LCD display panel, each color 
can be driven with any value included within a range of 0 to 255. Thus, if there is an attempt, 
either through some form of picture processing, or indeed user control, to increase the output 
to above the maximum allowed value, a clipping effect will result, causing all the driving 
levels above the threshold to be represented by the same value, i.e. the maximum saturation 

25 value. Thus, some picture detail which is quite clearly present in the input picture signal will 
therefore be lost through the amalgamations of the saturation levels caused by such a clipping 
effect. 
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It is an object of the invention to enhance images. The invention is defined by 
the independent claims. The dependent claims define advantageous embodiments. 

According to one aspect of the present invention, a method of non-linear 
processing of input picture signals is characterized in that the non-linear processing is 
responsive to hue values of the input picture signals. 

It is a particular advantage that the processing can serve to limit, or indeed 
avoid, any clipping of the output signal. This can be achieved by increasing the saturation up 
to a maximum level, at which clipping does not yet take place. Use is made of, amongst 
others, the hue values to determine this maximum level. 

The invention has the advantage that, for example, color saturation of the 
output picture signals can thereby be controlled in a maimer avoiding loss of picture detail, 
because the color saturation is increased more for picture areas showing low saturation levels 
than for picture areas showing high saturation levels. The clipping effect discussed above is 
therefore advantageously avoided. 

The set of input parameter values preferably comprises the saturation values, 
as an increase of the saturation considerably enhances the image to be displayed. 

Alternatively, adaptations of the brightness values can be applied. By suitably 
combining adaptations of saturation values as well as brightness values, there will be no 
undesired discoloration of the image. 

A simple method of non-linear processing is raising the parameter values to a 
certain power. By selecting a power depending on the hue values, the non-linear processing 
takes into account that the output picture signals show no (or a negligible) clipping effect. 

The non-linear processing can further be refined (in the sense of, for example, 
maximizing the saturation, while avoiding or minimizing clipping), by taking into account 
the maximum saturation value Smax during the non-linear processing of the saturation values. 

It is advantageous if the maximum saturation value Smax is dependent on the 
hue values or the output parameter values corresponding to the brightness (preferably 
depending on both hue values as well as on output brightness values). 

The powers used in the non-linear processing can also be made dependent on 
the content of the image, represented by a histogram of one or more sets of input parameter 
values. This is a further refinement to adapt the non-linear processing in such a way that, for 
example, saturation is further increased, while clipping effects are further minimized. 

In the case of input picture signals representing real-time video information, 
the video information can be considered as a sequence of images. Each image has its own 
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histogram. The powers can be adapted directly, depending on the content of the histogram, or 
alternatively, a more gradual change can be applied by adapting the powers based on an 
averaging over several images (low-pass filtering) of the changes of the powers. In this way, 
irregularities are avoided, which would arise from sudden large changes of the powers from 

5 image to image. 

Thus, it should be appreciated that, with the present invention, the saturation 
and/or the brightness of each pixel within a display can be modified in a non-linear manner 
that serves to introduce significant enhancements where the input picture signals have low 
saturation values, less significant enhancements where the saturation values of the input 

10 signals are close to their maximum limits, and no changes where the input signals already 
have maximum saturation values. 



These and other aspects of the invention are apparent from and will be 
1 5 elucidated with reference to the drawings in which: 

Fig. 1 is a basic block diagram of a system embodying the general concept of 
the present invention; 

Fig. 2 is a schematic block diagram showing the system of Fig. 1 in further 

detail; 

20 Fig. 3 is a schematic block diagram illustrating one particular implementation 

of the non-linear saturation processing block of Fig. 2; 

Fig. 4 is a schematic block diagram of the non-linear luminance processing 
block of Fig. 2; 

Fig. 5a illustrates a graphical representation between maximum saturation and 
25 brightness for a given hue; 

Figs. 5b to 5d illustrate suitable values for intermediate parameters. 

Figs. 6a and 6b comprise schematic block diagrams of a hardware 
implementation of the saturation bound evaluation block of Fig. 2; and 

Fig.7 illustrates suitable values for the power y h . 
30 Fig. 8 illustrates suitable values for the other power y Y . 

• 

Fig. 1 shows a generic block diagram illustrating a system 10 arranged to drive 
a display device in accordance with an embodiment of the present invention and which 
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includes a non-linear processing engine 12 arranged to receive red, green and blue input 
pictures signals 14 and, once processed, to output red, green and blue output picture signals 
16 for a display device such as an LCD panel. The operation of the non-linear processing 
engine 12 is controlled by means of a microcontroller 18 which is itself responsive to signals 
from a histogram evaluator 20 which samples the red, green and blue input picture signals 14 
or parameters derived from these signals (not shown). In this illustrated example, the 
histogram evaluator 20 works on brightness, saturation and hue values of the input picture 
signals, as will be further illustrated with reference to Fig. 2. Of course, the Figures illustrate 
merely one possible implementation of the present invention. The picture signals 14 represent 
an image composed of a matrix of rows and columns of pixels. The red, green and blue 
picture signals 14, 16 comprise information in a digital format about the red, green and blue 
color components of the pixels of an image. In the case of real-time video signals, these 
signals are handled as a sequence of images. The processing can be applied to each of these 
images. 

Fig. 2 illustrates, in somewhat greater detail, one possible embodiment of Fig. 
1 and each of the processing blocks illustrated therein will now be discussed in turn. 

A first converter block 22 serves to convert the Red R, Green G and Blue B 
input picture signals 14 into Brightness Y, Hue H, and Saturation S parameter values. 

In the illustrated embodiment, this block can be split functionally into two sub- 
blocks, i.e. a linear transformation accomplished through a matrix followed by a non-linear 
block. 

By means of the linear block, Brightness Y and chromaticity coordinates C r> 
C b are computed as follows for all pixels: 
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wherein the matrix ay comprises a predetermined set of parameters, which depend on the 
color coordinates of the display device to be driven. This computation can be executed in 
hardware by using three multipliers and three adders for each row. 

The saturation S and hue values H for a pixel are then defined by the following 

formulae: 
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S=(C r 2 + C b 2 )* 
H=tan (Q/C b ). 

These functions can be implemented in several ways, for example, by using 
Look Up Tables (LUTs) and/or hardware interpolators. 

A non-linear saturation processing block 24 is arranged to receive maximum 
saturation values S max and the saturation values from the converter 22. The way in which the 
maximum saturation value Sma X is determined for a pixel will be explained hereinafter. In the 
illustrated embodiment, a non-linear function in the form of a gamma function is employed 
as follows: 

wherein S' is the output parameter of the output picture signals 16 corresponding to the 
saturation and y h is the power used in this non-linear processing. 

The transfer function can be rearranged in this form: 

S' = 2 Yh l ,og 2 ( s )" !og ^ s «« M +log2 ( s »« ) 

This function can be implemented in several ways, for example, using LUTs 
and multipliers. One possible embodiment is illustrated in Fig. 3. 

First, the logarithm of the saturation S and maximum saturation value is 
determined in respective logarithmic blocks 302, 301. Then, the results are subtracted in 
subtractor 303. The output of the subtracter 303, being Log 2 (S) - Log 2 (S^ ) , is multiplied 
by the power y h in multiplier 304. The output of the multiplier 304 is added to the output 
Log 2 (S^ ) of block 301 in adder 305. The output of the adder 305 is used as power in 
exponential block 306, wherein the number two is raised to this power. The result, present at 
the output of the exponential block 306, is the output saturation value S\ 

A saturation bound evaluation block 34 receives both the hue values H from 
the converter 22 and the output brightness value Y' from the luminance-processing block 32. 
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The graph of Fig. 5a illustrates a typical relationship between the maximum 
saturation values and the brightness values Y for a given Hue value H. Intermediate 
parameters m, , m 2 and Y, are functions of the hue values H. These values can be readily 
obtained from a LUT. Intermediate parameters mi and m 2 are defined as a tangent of the 
respective angles a, and as shown in Fig. 5a; 

mi = tg (ai) 
m 2 = tg (a*) 

In Fig. 5b, 5c, 5d suitable values for the respective intermediate parameters 
m, , m 2 and Y t are provided. These suitable values have experimentally been found in a 
situation, where the input parameter values are available in an eight-bit digital format 
(resulting in 256 possible different values) and the matrix ajj comprises the following values: 

an = 66 ai2 = 129 ai 3 = 25 

a 2 i = -38 a 22 = -74 a 23 = 1 12 

a 3 i = 112 a 32 = -94 a 33 = -18 

When applying this matrix ajj, the minimum brightness value Y„u n (as shown in 
Fig. 5a) is equal to zero, while the maximum brightness value Ymax amounts to 220. 

A block diagram of a possible hardware implementation of the saturation 
bound evaluation block 34 is given in Fig. 6a. A first branch computes the function 
(Y - Y').m 2 by means of first subtracter 601 and first multiplier 602. A second branch 
computes the function (Y'-Y,^ )m, using second subtracter 603 and second multiplier 604. 
Comparator 605 determines whether Y* is larger or smaller than Y,. If Y' is larger than Y, 
multiplexer 606 selects (Y,,^ - Y')m 2 as output. If Y* is smaller than Y, , multiplexer 606 
selects (Y'-Y^,, )m, as output. In this way, the output of the multiplexer 606 corresponds to 
the desired maximum saturation values Smax as a function of the output brightness values Y' 
as shown in Fig. 5a. 

The intermediate parameters mi, m 2 and Y t are obtainable from LUTs, which 
receive the hue value H as input (not shown in Fig. 6a). These LUTs contain the intermediate 
parameters as shown in the graphs of Figs. 5b, 5c, 5d. 



WO 2004/008778 PCT/IB2003/003118 

7 

An alternative block diagram of a possible hardware implementation of the 
saturation bound evaluation block 34 is given in Fig. 6b. The comparator 610 determines 
again whether Y' is larger than Y t : 

If not, the multiplexers 61 1, 612, 615 select as output the input values present 
at the input indicated by the dotted lines inside the multiplexers 611,612, 615. 

If yes, the multiplexers 611, 612, 615 select the input values present at the 

other inputs. 

As a result, (Y'-Y^ ) , or (Y^ - Y'), respectively, will be present at the 

output of the subtracter 613. 

The multiplier 614 multiplies the mentioned outputs of subtracter 613 by mi, 
m2> respectively, depending on the output of comparator 610. 

The multiplexer 615 selects mi, m 2 , respectively, depending on the output of 
comparator 610. The resulting output of the multiplier 614 is again the desired maximum 
saturation value Smax* 

S max =(Y'-Y min )m I ,ifY^Y t 

S m »=(Y man -Y')m 2 ,ifY l >Y, 

As mentioned before, the intermediate parameters are obtainable from LUTs. 

A y h LUT block 26 receives the hue values H from the converter 22. The y h 
value can comprise a function of hue. In this way, it is possible to apply different color 
saturation processing depending on the hue values H. This can be realized easily by using a 
LUT. An example of suitable values of the power y h as a function of the Hue values H is 
given in Fig. 7. The power y h shows larger values (close to one) within a limited range of 
hue values H. This limited range includes, for example, the hue values H around 145 
corresponding to skin colors. The human eye is very sensitive to these colors and an increase 
of saturation would result in "unnatural" images. By reducing the degree of non-linear 
processing (by selecting powers closer to one) for these skin colors, this problem is avoided. 
Outside the limited range, a considerable "boosting" of the colors is obtained by applying, for 
example, a value of 0.7 for y h . 

Turning back to Fig. 2, a non-linear luminance processing block 32 is arranged 
to receive the brightness value Y from the converter 22. 

In this embodiment, a non-linear function, using another power y Y is applied 
wherein the output brightness value Y 1 is provided by the function: 
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Y f = Y Yv 



The transfer function can be rearranged in the following form: 

5 

Y'= 2 yY l ,og2 Y l 

The above function can be realized, for example, as shown in the hardware 
block diagram of Fig. 4. The logarithm of the input brightness value Y is determined in the 

10 logarithmic block 401 . The output of the logarithmic block 401 is multiplied by the other 

power y Y in multiplier 402. The output of multiplier 402 is used as power in the exponential 
block 403 to raise the number two to this power. The result present at the output of the 
exponential block 403 is the output brightness value Y 1 . An alternative for the above 
hardware block diagram is to apply LUTs. 

1 5 In Fig. 2, a y Y LUT Block 28 also receives the hue values H from the 

converter 22. The y Y value can also comprise a function of hue, and so it is possible to apply 
different brightness processing depending on the hue values H. This can be easily realized by 
way of an appropriate LUT. An example of suitable values of the other power y Y as a 
function of the hue values H is given in Fig. 8. The other power y Y shows a value close to 

20 one within a limited range of the hue values H (around 145). Again, in this range, the degree 
of non-linear processing is reduced (or made zero) in order to avoid unnatural skin colors. 
Outside this range, the other power y Y preferably has a value larger than one, for example 
1.2. 

In Fig. 2, a YHS to RGB converter 30 comprises the last block in the signal 
25 flow path and serves to convert the output parameter values Y',S\H to the output picture 
signals R'G'B'. 

This block can be implemented by cascading a non-linear transformation 
function for obtaining output chromaticity coordinates Cr f and Cb* from the output 
saturation values S' and hue values H, followed by a linear transformation converting these 
30 coordinates in output picture signals R f , G 1 and B 1 . 

The formulae for computing output coordinates Cr f and Cb 1 are: 



'1 
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Cb'=S'cos(H) 
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The functions sin(H) and cos(H) can be implemented through Look Up Tables 
and interpolators. 

5 Furthermore, the values of output picture signals R\G\ B' can be computed 

by multiplying Y 1 , C r 1 , C b 1 by an appropriate matrix by as shown below. 
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1 o wherein the matrix by comprises a predetermined set of parameters depending 

on the color coordinates of the display device. The function requires three multipliers and 
three adders for each matrix row. 

The non-linear processing can also be made dependent on the distribution of 
the input parameter values Y, S, H of an image. This distribution can be evaluated by using a 
1 5 histogram evaluator 20 as shown in Fig. 2. The evaluator 20 receives the input parameter 
values Y, S, H and determines the number of occurrences of combinations (triplets) of input 
values Y, H, S in an area (a "window") of an image (or in the whole image). 

Depending on the distribution the powers y Y and y h can be adapted in the y Y 
LUT 28 and y h LUT 26, respectively, via micro-controller 18. As a result, the degree of non- 
20 linear processing can be adapted in dependence on the distribution. Since the main goal is to 
adapt the color saturation of an area of an image for a given hue value H, a suitable parameter 
to use is the average saturation value S (Y)| H , which is a function of the input brightness 
value Y. 

This average saturation value S(Y)| H as a function of input brightness values 
25 Y can be derived from the histogram Q (Y, H, S) as follows: 



S(Y) 



h = E S i O|„(Y,S i ), 



wherein Q H is the histogram for a given hue value H, i is an index for the range of possible 
input saturation values S 1 and S, is the i-th possible input saturation value. In the case of 8- 
bit digital input picture signals, the range of i is 256. 
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A refinement can be applied by taking into account a weighing factor 
W (Y, H) , that weighs the average saturation values S(Y)| H depending on the maximum 

possible saturation values for a given input brightness value Y. 

Taking into account the weighing factor W(Y,H), the weighed average 
saturation value becomes: 

S'(Y)|„=W(Y,H)S(Y)| H =- S(Y) ' H 



As a subsequent step, a parameter \|/| H , being a weighed average hue 
saturation value for a given hue value, can be computed from the expression: 



10 v|/ 



H =XY j S'(Y j )|„, 



J 



wherein j is an index for the range of possible input brightness values Y. 

The weighed average hue saturation value \|/| H related to an area of an image 
can be used to adapt the degree of non-linear processing by adapting the power y h . 

In the case of video information with changing areas in an image or changing 
1 5 images, the value \|/| H is determined for each image of the video information. The current 
value V|/| H is compared with a user-adjustable target value vj/ x | H . 

If the current value \\f\ H deviates from the target value y T \ H , an error signal, 
proportional to the difference of the 2 values, is used to adapt the power y h . By applying a 
low-pass filtering on the error signal, abrupt changes of the power y h due to changing video 

20 information are avoided. 

In this way, disturbing effects are avoided, when the output picture signals are 
displayed on the display device. 

It should be noted that the above-mentioned embodiments illustrate rather than 
limit the invention, and that those skilled in the art will be able to design many alternative 

25 embodiments without departing from the scope of the appended claims. In the claims, any 
reference signs placed between parentheses shall not be construed as limiting the claim. Use 
of the verb "to comprise" and its conjugations does not exclude the presence of elements or 
steps other than those stated in a claim. Use of the article "a" or "an" preceding an element 
does not exclude the presence of a plurality of such elements. The invention can be 
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implemented by means of hardware comprising several distinct elements, and by means of a 
suitably programmed computer. In the device claim enumerating several means, several of 
these means can be embodied by one and the same item of hardware. The mere fact that 
certain measures are recited in mutually different dependent claims does not indicate that a 
combination of these measures cannot be used to advantage. 



